Electronic Device with Adaptive Lighting System

ABSTRACT

A head-mounted device may include a display that generates content and an optical system through which the content is viewable. The head-mounted device may include a lighting system that illuminates a periphery of the optical system. When the user places the device on his or her head in a brightly lit environment, control circuitry may operate the lighting system to provide bright illumination to the user&#39;s peripheral vision. The lighting system may gradually decrease in brightness until the user transitions from a bright-adapted state to a dark-adapted state. When the user is partially or fully dark-adapted, the lighting system may be turned off and the display may be turned on. In some arrangements, an ambient light sensor may measure ambient light conditions outside of the electronic device and the control circuitry may control the lighting system based on the ambient lighting conditions.

This application claims the benefit of provisional patent applicationNo. 62/734,703, filed Sep. 21, 2018, which is hereby incorporated byreference herein in its entirety.

BACKGROUND

This relates generally to displays and, more particularly, to displaysfor head-mounted devices.

Head-mounted devices such as virtual reality glasses and augmentedreality glasses use displays to generate images for a user.

If care is not taken, a head-mounted device may be cumbersome and tiringto wear. The images on the display may appear too dark and washed outwhen the user first puts the head-mounted device on his or her head. Theuser may experience dazzle or discomfort when transitioning out of avirtual reality viewing experience. The dynamic range of a head-mounteddisplay may be perceived as insufficient depending on the adaptationstate of the user's eyes.

SUMMARY

A head-mounted electronic device configured to be worn on a user's headmay include a display that generates display content and an opticalsystem through which the display content is viewable. The head-mounteddevice may include a lighting system that illuminates a periphery of theoptical system. When the user places the device on his or her head in abrightly lit environment, control circuitry may operate the lightingsystem to provide bright illumination to the user's peripheral vision.The lighting system may gradually decrease in brightness until the usertransitions from a bright-adapted state to a dark-adapted state. Whenthe user is partially or fully dark-adapted, the lighting system may beturned off and the display may be turned on. Conversely, when a user isabout to remove the device from his or her head, the control circuitrymay gradually increase the brightness of the lighting system so that theuser can transition from a dark-adapted state to a bright-adapted statebefore removing the device.

In some arrangements, an ambient light sensor may measure ambient lightconditions outside of the electronic device and the control circuitrymay control the lighting system based on the ambient lightingconditions.

Control circuitry in the electronic device may estimate a brightnessadaptation state of the user that is wearing the electronic device. Thecontrol circuitry may adjust a brightness of the lighting system basedon the user's adaptation state. This may include, for example, adjustingthe brightness of the lighting system based on ambient light conditions,physiological attributes of the user, motion sensor data, gaze position,and/or other information.

The lighting system may include one or more light sources. The lightsources may be light-emitting diodes. The control circuitry may beconfigured to independently control the brightness and/or color of eachlight source (e.g., some light sources may be turned off while othersare turned on, some light sources may have one brightness and others mayhave a different brightness, etc.).

In some arrangements, the light sources may include red, green, and bluelight-emitting diodes or light sources of other colors so that controlcircuitry can adjust a color of illumination from the lighting system.The color of illumination from the lighting system may, for example, beadjusted to match or more closely match the color of ambient light sothat the transition from the ambient light to the head-mounted displaylight is less abrupt.

The lighting system may include a light source that emits light into alight guide. The light guide may guide the light around a periphery ofthe optical system via total internal reflection. The light guide mayhave light extraction features that allow the light to escape from thelight guide.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an illustrative head-mounted device in accordancewith an embodiment.

FIG. 2 is a graph showing how content luminance may be mapped to displayluminance based on a variety of different brightness adaptation statesin accordance with an embodiment.

FIG. 3 is a perspective view of an illustrative head-mounted devicehaving a lighting system that illuminates a periphery of an opticalsystem in accordance with an embodiment.

FIG. 4 is a front view of an illustrative head-mounted device having alighting system that includes a light source and a light guide inaccordance with an embodiment.

FIG. 5 is a diagram showing how control circuitry may use informationfrom sensors and other input-output devices to determine operatingconditions for a lighting system and to determine tone mappingparameters for a display in accordance with an embodiment.

FIG. 6 is a graph showing how a lighting system may be used to help auser transition from a bright-adapted state to a dark-adapted stateafter a user places a head-mounted device on his or her head inaccordance with an embodiment.

FIG. 7 is a graph showing how a lighting system may be used to help auser transition from a dark-adapted state to a bright-adapted statebefore the user removes a head-mounted device from his or her head inaccordance with an embodiment.

DETAILED DESCRIPTION

Head-mounted devices such as head-mounted displays may be used forvirtual reality and augmented reality systems. For example, a pair ofvirtual reality glasses that is worn on the head of a user may be usedto provide a user with virtual reality content.

An illustrative system in which a head-mounted device such as a pair ofvirtual reality glasses is used in providing a user with display contentsuch as virtual reality content is shown in FIG. 1. As shown in FIG. 1,head-mounted device 10 may include a display system such as displaysystem 40 that creates images and may have an optical system such asoptical system 20 through which a user (see, e.g., user's eyes 46) mayview the images produced by display system 40 in direction 48. Opticalsystem 20 may, for example, include a first lens module (e.g., for auser's left eye) and a second lens module (e.g., for a user's righteye).

Display system 40 may be based on a liquid crystal display, an organiclight-emitting diode display, a display having an array of crystallinesemiconductor light-emitting diode dies, a liquid-crystal-on-silicondisplay, a microelectromechanical systems (MEMs) display, and/ordisplays based on other display technologies. Separate left and rightdisplays may be included in system 40 for the user's left and right eyesor a single display may span both eyes.

Visual content (e.g., image data for still and/or moving images) may beprovided to display system 40 using control circuitry 42 that is mountedin head-mounted device 10 and/or control circuitry that is mountedoutside of head-mounted device 10 (e.g., in an associated portableelectronic device, laptop computer, or other computing equipment).Control circuitry 42 may include storage such as hard-disk storage,volatile and non-volatile memory, electrically programmable storage forforming a solid-state drive, and other memory. Control circuitry 42 mayalso include one or more microprocessors, microcontrollers, digitalsignal processors, graphics processors, baseband processors,application-specific integrated circuits, and other processingcircuitry. Communications circuits in circuitry 42 may be used totransmit and receive data (e.g., wirelessly and/or over wired paths).Control circuitry 42 may use display system 40 to display visual contentsuch as virtual reality content (e.g., computer-generated contentassociated with a virtual world), pre-recorded video for a movie orother media, or other images.

During operation, a content generator in device 10 such as contentgenerator 12 (e.g., operating system functions and/or applicationsrunning on control circuitry 42) may generate content for display system40 (e.g., virtual reality content, high dynamic range content, standarddynamic range content, etc.). A luminance value mapping circuitry suchas tone mapping circuitry 14 may be used to provide content generatorswith tone mapping parameters (sometimes referred to as luminance valuemapping parameters) indicating how the content generators should mapcontent luminance values to display luminance values and/or may be usedto directly perform content-luminance-to-display-luminance mappingoperations on content luminance values from the content generators. Forexample, tone mapping circuitry 14 may produce tone mapping parametersthat are based on the current adaptation level of the user's visualsystem to use in producing display luminance values for use indisplaying images with display system 40. Tone mapping circuitry 14 maybe implemented using code running on control circuitry 42 and/or othercontrol circuitry and/or may use hardwired features of the controlcircuitry in device 10. The tone mapping parameters may be expressed inany suitable format. For example, tone mapping parameters such asadaptation level, black level, reference white level, and/or specularwhite level may be expressed in cd/m².

The human visual system is capable of perceiving a large range ofluminance levels. However, the human eye cannot see all of thesebrightness levels at the same time. Rather, the eye continuously adjustsits sensitivity to the viewing environment in order to perceivedifferent ranges of luminance levels within the eye's larger overalldynamic range. The current sensitivity level of the human visual systemis sometimes referred to as its brightness adaptation level. Thesubjective brightness perceived by a user is usually dependent on theuser's brightness adaptation level. When the human visual system isbright-adapted (i.e., adapted to bright light), the eye is lesssensitive. In contrast, when the human visual system is dark-adapted(i.e., adapted to dim light), the eye is more sensitive.

If care is not taken, a user that is adapted to bright ambient lightbefore using a head-mounted display may initially perceive the images onthe display as dim and washed out until the user's eyes adapt to thedarkness of the head-mounted display. Conversely, a user that is adaptedto the darkness of a head-mounted display may experience dazzle anddiscomfort when the user removes the head-mounted display and is facedwith bright ambient light.

To enhance the user's experience with head-mounted device 10, controlcircuitry 42 may be configured to determine an adaptation state of theuser. A user's adaptation state may be represented by a luminance valueor a range of luminance values. Control circuitry 42 may use tonemapping circuitry 14 to adapt display data according to the adaptationstate of the user. This may include, for example, matching (e.g.,optimizing) a brightness range of display 40 with the current adaptationstate of the user, adjusting a brightness range of display 40 to have adesired effect on the user's adaptation state (e.g., to help “guide” theuser's current adaptation state to a different adaptation state),adjusting a brightness range at certain periods of time to boost theperceived brightness or darkness at another time, adjusting brightnessfor some portions of an image to boost the perceived brightness ordarkness of other portions of an image, selecting appropriate tonemapping parameters based on the adaptation state of the user, and/ortaking other actions based on the estimated adaptation state of theuser. Tone mapping circuitry 14 may be configured to adapt display datafor left and right displays. The display data adjustment may be the samefor both displays or the display data adjustment may be different forthe left and right displays. For example, tone mapping circuitry 14 mayuse different tone mapping curves for the left and right displays toaccount for different tone mapping needs for the user's left and righteyes, if desired.

In some arrangements, the dynamic range of display 40 may not be largeenough to match the user's brightness adaptation state. For example, ina bright outdoor environment, the user's brightness adaptation level maybe greater than 20,000 nits. In an overcast outdoor environment orindoors near a bright window, the user's brightness adaptation level maybe between 1,000 nits and 3,000 nits. In a bright indoor environment,the user's brightness adaptation level may be between 300 nits and 500nits. In some of these scenarios, the maximum brightness of display 40may be less than the brightness adaptation level of the user.

To help device 10 accommodate the adaptation state of the user, device10 may include lighting system 60. Lighting system 60 may includelight-emitting devices 62. Light-emitting devices 62 may be locatedaround the periphery of display 40 and/or optical system 20, abovedisplay 40 and/or optical system 20, below display 40 and/or opticalsystem 20, behind display 40 and/or optical system 20, to the leftand/or right of display 40 and/or optical system 20, and/or in otherlocations of device 10. When device 10 is mounted on a user's head,lighting-emitting devices 62 may produce light in the user's peripheralvision.

Lighting system 60 may include any suitable light sources that producelight in response to applied electrical signals such as lamps,light-emitting diodes, lasers, arrays of light sources, individual lightsources, backlit or edge-lit light guides, light sources that emit oneor more beams of light (e.g., a laser beam, light-emitting diode beam,or a beam associated with another collimated light source), lightsources that emit light in a fixed pattern of one or more beams, lightsources that emit light using raster scanning techniques, light sourcesthat emit steerable beams (e.g., light sources with mirror arrays tosteer light in a light projector system, light sources with one or moresteerable mirrors, steerable lasers and light-emitting diodes, etc.),light guide panels and/or pipes that contain light extraction featuresthat cause the light guide panels and/or pipes to emit light in variouspatterns, and other electrically controlled light sources.

Instead of or in addition to including light sources 62, lighting system60 may include one or more windows that selectively allow ambient lightfrom the exterior of device 10 to reach the interior of device 10 tohelp provide a smooth transition from bright lighting conditions to darklighting conditions and vice versa. For example, lighting system 60 mayinclude one or more apertures through which ambient light may reach theinterior of device 10. The apertures may have an adjustable size to letin adjustable amounts of ambient light, may include adjustable filtersthat can adjust the transparency of the apertures, and/or may beprovided with shutters that can selectively open and close theapertures. Control circuitry 42 may be used to control the amount ofambient light that passes through the apertures. This is, however,merely illustrative. If desired, lighting system 60 may include lightsources 62 and may not include any apertures.

If desired, light-emitting devices 62 may include colored light sources.For example, devices 62 may include red, green, and blue light-emittingdiodes, thereby allowing system 60 to emit light of an adjustable colorby adjusting the relative strengths of the red, green, and blue lightemitted from devices 62. If desired, light-emitting devices 62 mayinclude blue light-emitting diodes with different wavelengths. A firstwavelength of blue light may tend to suppress a user's melatoninproduction whereas a second wavelength of blue light may have little tono effect on the user's melatonin production. Control circuitry 42 maycontrol the relative brightness of the two blue light sources in system60 to have a desired effect on the user's circadian rhythm (e.g., byusing the first wavelength of blue light in the morning and using thesecond wavelength of blue light in the evening, for example).

Lighting system 60 may contain individually controlled areas. Theseareas may be relatively small areas (e.g., pixel-sized areas) and/or maybe larger areas. For example, lighting system 60 may containlight-producing devices that produce a single block of light over anentire periphery of device 10. If desired, lighting system 60 mayinclude one or more light diffusion layers (e.g., frosted polymer films,films or substrates with light-scattering particles, etc.).

Lighting system 60 may, if desired, include light guiding structuressuch as optical fibers and/or other light guiding elements. Light guidesin system 60 may be used to guide light from a light source to alocation in device 10 where illumination is desired. Light guides insystem 60 may, in some arrangements, be transparent and may blend inwith a surrounding portion of device 10 (e.g., to generate a uniformappearance).

Control circuity 42 may control lighting system 60 based on theadaptation state of the user. This may include, for example, matching abrightness range of lighting system 60 with the current adaptation stateof the user, adjusting a brightness range of lighting system 60 to havea desired effect on the user's adaptation state (e.g., to help “guide”the user's current adaptation state to a different adaptation state),adjusting a brightness range of lighting system 60 at certain periods oftime to boost the perceived brightness or darkness at another time,adjusting brightness of lighting system 60 for a subset oflight-emitting devices 62 near one portion of an image on display 40 toboost the perceived brightness or darkness of other portions of an imageon display 40, adjusting a color of light emitted by lighting system 60to match the color of ambient light or to enhance appearance of colorson display 40, and/or taking other actions based on the estimatedadaptation state of the user.

Input-output devices 18 may be coupled to control circuitry 42.Input-output devices 18 may be mounted in head-mounted device 10 and/ormay be mounted outside of head-mounted device 10 (e.g., in an associatedportable electronic device, laptop computer, or other computingequipment). Input-output devices 18 may be used to gather user inputfrom a user, may be used to make measurements on the environmentsurrounding device 10, may be used to provide output to a user, and/ormay be used to supply output to external electronic equipment.Input-output devices 18 may include buttons, joysticks, keypads,keyboard keys, touch sensors, track pads, displays, touch screendisplays, microphones, speakers, light-emitting diodes for providing auser with visual output, and sensors (e.g., force sensors, temperaturesensors, magnetic sensor, accelerometers, gyroscopes, and/or othersensors for measuring orientation, position, and/or movement of glasses10, proximity sensors, capacitive touch sensors, strain gauges, gassensors, pressure sensors, ambient light sensors, and/or other sensors).

Input-output circuitry 18 may include a color ambient light sensor orother ambient light sensor 22 for gathering ambient light measurements(e.g., ambient light levels such as ambient light luminance measurementsand/or ambient light color measurements such as color temperaturemeasurements and/or color coordinate measurements). Input-outputcircuitry 18 may also include cameras 26 (digital image sensors) forcapturing images of the user's surroundings, for performing gazedetection operations by viewing eyes 46, and/or other cameras.Input-output devices 18 may include a sensing system that measurescharacteristics of the user's eyes 46. For example, light source 26 andcamera 24 may be used in supplying light to eye 46 and measuringreflected light to measure the optical properties of eye 46. Lightsource 26 may produce light at any suitable wavelength (e.g., nearinfrared light wavelengths, longer infrared wavelengths, visiblewavelengths, etc.). Camera 24 and/or light source 26 may be used indetermining pupil size, blink rate, facial expression, eye openness(e.g., whether the user is squinting), etc. Camera 24 may also be usedby control circuitry 42 to gather images of the pupils and otherportions of the eyes of the viewer. The locations of the viewer's pupilsand the locations of the viewer's pupils relative to the rest of theviewer's eyes may be used to determine the locations of the centers ofthe viewer's eyes (i.e., the centers of the user's pupils) and thedirection of view (gaze direction) of the viewer's eyes.

Input-output device 18 may include one or more motion sensors 92. Motionsensors 92 may include one or more accelerometers, compasses,gyroscopes, barometers, pressure sensors, magnetic sensors, inertialmeasurement units that contain some or all of these sensors, and/orother sensors for measuring orientation, position, and/or movement ofdevice 10. Motion sensors 30 may produce sensor data that indicateswhether device 10 is being place on or removed from a user's head. Forexample, an upward motion arc or lifting from a surface may indicatethat device 10 is being placed on or have been placed on a user's head,whereas a downward motion arc or setting down onto a surface mayindicate that device 10 is being removed from or has been removed from auser's head.

FIG. 2 is a graph showing how content luminance values for display 40can be mapped to display luminance values for display 40 by device 10for three different illustrative adaptation states. The contentluminance and display luminance axes of the graph of FIG. 2 havelogarithmic scales. In the example of FIG. 2, curve 28 is optimized fora dark adaptation state (e.g., when the user is adapted to a darkenvironment), curve 30 is optimized for a moderate adaptation state(e.g., when a user is adapted to a lit office or a dim outdoorsenvironment), and curve 32 is optimized for a bright adaptation state(e.g., when a user is adapted to a bright outdoors environment). Theremay be greater or fewer than three different optimized curves formapping content luminance values to display luminance values based onthe adaptation state of the user. The use of three curves each optimizedfor one of three adaptation states is merely illustrative.

Tone mapping circuitry 14 may switch from one tone mapping curve toanother tone mapping curve based on various factors. In some scenarios,tone mapping circuitry 14 may select the tone mapping curve that isoptimized for the user's current adaptation state. For example, when auser first places device 10 on his or her head, the user's eyes maygradually adapt from the bright room environment to the dark viewingconditions of device 10. Tone mapping circuitry 14 may follow the user'sadaptation in this scenario, using bright adaptation curve 32 at firstand gradually moving to dark adaptation curve 28.

Control circuitry 42 may also control lighting system 60 to follow theuser's adaptation state. For example, when the user first puts device 10on his or her head in a bright room, lighting system 60 may start at afirst luminance level (e.g., matching or nearly matching the ambientbrightness level) and may gradually shift to a second luminance level.The second luminance level may be less than the first luminance level.If desired, lighting system 60 may be turned off after the user isdark-adapted to the interior of device 10 (i.e., the second luminancelevel may be zero).

If desired, control circuitry 42 may also adjust the color temperatureof lighting system 60. For example, when the user first puts device 10on his or her head in an environment with cool ambient lighting (e.g.,bluish ambient lighting), lighting system 60 may start at a first colortemperature (e.g., a first color temperature corresponding to a coolwhite that matches or nearly matches the color of ambient light) and maygradually shift to a second color temperature. The second colortemperature may be different from the first color temperature. Forexample, the second color temperature may be lower than the first colortemperature (e.g., the second color temperature may correspond to a warmwhite). This is, however, merely illustrative. If desired, controlcircuitry 42 may only adjust the brightness of lighting system 60without adjusting the color temperature of lighting system 60.

In some scenarios, tone mapping circuitry 14 and/or lighting system 60may be used to drive or guide a user's adaptation from its current stateto a different state. For example, when a video that the user iswatching ends, when the user enters a home screen, or when a userotherwise indicates that he or she is transitioning out of the virtualreality experience, tone mapping circuitry 14 may adjust display dataand control circuitry 42 may control lighting system 60 to graduallyshift the user's adaptation state from dark-adapted to bright-adapted toavoid any dazzle or discomfort when the user removes device 10. In thistype of scenario, tone mapping circuitry 14 may use dark adaptationcurve 28 at first and gradually move to bright adaptation curve 32,thereby causing the user's eyes to slowly become bright-adapted.

In some scenarios, control circuitry 42 may control the brightnessand/or color temperature of lighting system 60 to enhance the images ondisplay 40. For example, lighting system 60 may produce magenta lightingto enhance the appearance of green colors on display 40. As anotherexample, a first set of light-emitting devices 62 in lighting system 60may be illuminated near a first portion of an image on display 40 whilea second set of light-emitting devices 62 in lighting system 60 near asecond portion of an image on display 40 may be left unilluminated,which in turn may result in a perceived contrast boost in the secondportion of the image.

FIG. 3 is a perspective view of an illustrative device 10 having alighting system such as lighting system 60. Device 10 may include innerand outer surfaces such as outer surface 16 and inner surface 64. Innersurface 64 may face the user's eyes 46 when the user is wearing device10 and viewing displays 40 (not shown in FIG. 3) through optical system20 in direction 48. Ambient light sensor 22 may be configured to measurethe brightness and/or color of ambient light outside of device 10.

Lighting system 60 may include light-emitting devices 62 on innersurface 64 of device 10. Light-emitting devices 62 may emit light 66towards the user's peripheral vision. In the example of FIG. 3,light-emitting devices 62 are distributed in a loop surrounding theperiphery of optical system 20. This is merely illustrative. If desired,light-emitting devices 62 may be distributed in a first loop surroundinga left-eye portion of optical system 20 and a second loop surrounding aright-eye portion of optical system 20. If desired, light-emittingdevices 62 may be located elsewhere in device 10. There may be one, two,three, four, ten, twenty, fifty, more than fifty, or less than fiftylight-emitting devices 62 in system 60.

Control circuitry 42 may control all of light-emitting devices 62 inunison or may control all or some of light-emitting devices 62independently of one another. For example, control circuitry 42 may setone or more light-emitting devices 62 (e.g., light-emitting devices 62in region 70) at a first brightness level while setting otherlight-emitting devices 62 at a second brightness level different fromthe first brightness level.

The example of FIG. 3 in which light-emitting devices 62 emit light 66directly out of device 10 towards a user's peripheral vision is merelyillustrative. If desired, light-emitting devices 62 may emit light intoa light guide (e.g., a light guide pipe or light guide panel) thatguides the light to a different location via total internal reflection.This type of arrangement is illustrated in FIG. 4.

As shown in FIG. 4, light-emitting device 62 may emit light 66 into alight guide such as light guide 68. Light guide 68 may be a fiber, amolded plastic structure, or other light guiding structure that guideslight internally in accordance with the principle of total internalreflection. Light guide 68 may be formed from clear plastic, glass,sapphire or other transparent crystalline materials, or othertransparent materials. In some configurations, light guides 68 may haveinner structures (sometimes referred to as cores) that are coated withone or more outer layers (sometimes referred to as claddings or coatinglayers). In this type of arrangement, the core may have a higher indexof refraction than the cladding to promote total internal reflection ofthe light that has been coupled into light guide 68. High/low index ofrefraction arrangements may also be created by embedding a light guidestructure of a first index of refraction into a transparent material ofa second index of refraction that is higher than the first index ofrefraction. The transparent material into which the light guidestructure is embedded may be a polymer or other clear binder.

In general, light guides such as light guide 68 may be formed byinjection molding, by machining plastic light guide structures, bydipping or spraying polymer coatings onto machined or molded plasticcore parts or glass core parts, by extruding polymers, by elongatingglass or plastic rods using heat and tension, or by otherwise formingstructures that can internally guide light within device 10. With onesuitable arrangement, which may sometimes be described herein as anexample, light guide 68 is an optical fiber having a circularcross-sectional shape with a central core surrounded by a cladding layerof lower index of refraction material. Light guide 68 may be formed formglass, plastic, or other transparent material. Arrangements in whichlight guide 68 has a non-circular cross-sectional shape may also beused.

In regions of light guide 68 where illumination is desired, light guide68 may have light extraction features such as particles, changes inrefractive index, roughened surfaces, protrusions such as bumps orridges, recesses such as pits or grooves, or other light extractionfeatures. In the presence of light extraction features in light guide68, light 66 from the interior of light guide 68 may be scattered out oflight guide 68. If desired, some portions of light guide 68 may be freeof light extraction features so that light 66 propagates through and iscontained within that portion of light guide 68 via total internalreflection.

FIG. 5 is a diagram showing how control circuitry 42 may use informationfrom input-output devices 18 and/or other devices to determine operatingconditions for lighting system 60 and to determine tone mappingparameters for display 40. Control circuitry 42 may receive inputs suchas ambient light conditions 34, physiological attributes 36 of a user,motion sensor data 90, gaze position 38, and display content (e.g.,display content 44 and/or remapped content 50). Control circuitry 42 maycontrol lighting system 60 based on one or more of these inputs.

Ambient light conditions 34 may include an ambient light brightnessand/or an ambient light color measured with ambient light sensor 22. Themeasured ambient light information 34 may be indicative of the user'sadaptation state when the user puts device 10 on his or her head.Control circuitry 42 may control lighting system 60 based ambient lightinformation 34. For example, if ambient light sensor conditions 34indicate bright ambient light, control circuitry 42 may illuminatelighting system 60 with bright light so that images on display 40 appearsufficiently bright to the user when display 40 is turned on. Controlcircuitry 42 may gradually reduce the brightness of lighting system 60as the user adapts to the dark viewing conditions of device 10.

Control circuitry 42 (e.g., tone mapping circuitry 14 of FIG. 1) mayalso determine tone mapping parameters for display 40 based on ambientlight conditions 34. For example, if ambient light sensor conditions 34indicate bright ambient light, tone mapping circuitry 14 may begin withbright-adapted curve 32 so that the images appear sufficiently bright tothe user. Tone mapping circuitry 14 may gradually shift to tone mappingparameters that are optimized for a dark adaptation state as the useradapts to the dark viewing conditions of device 10.

Physiological attributes 36 may include a pupil size, blink rate, facialexpression, eye openness, and/or other information that may indicate theuser's adaptation state. For example, as a user's eyes adapt to darkambient conditions, the pupils may become larger, the blink rate maydecrease, a distance between eyes and cheeks may decrease as the facerelaxes, and eyes may squint less. These physiological attributes may bemeasured (e.g., using camera 24 and/or light emitter 26) and used bycontrol circuitry 42 to determine operating conditions for lightingsystem 60 and to determine tone mapping parameters for display 40. Inaddition to being indicative of a user's adaptation state, physiologicalattributes 36 may also be indicative of the user's fatigue level. Usersmay experience fatigue after viewing images on a head-mounted displayfor extended periods of time. If desired, control circuitry 42 may usephysiological attributes 36 to determine a fatigue level of the user andmay adjust lighting system 60 and/or display 40 accordingly. Forexample, control circuitry 42 may darken lighting system 60 and/or maydarken display 40 when a user is showing signs of a fatigue to make theviewing experience more comfortable.

Control circuitry 42 may use motion sensor data 90 from motion sensor 92of FIG. 1 to determine operating conditions for lighting system 60and/or to determine tone mapping parameters for display 40. Motionsensor data 90 may, for example, be used to determine when device 10 isbeing placed on or removed from a user's head. When motion sensor data90 indicates that device 10 is being placed on a user's head, controlcircuitry 42 may turn on lighting system 60 so that the user'stransition from bright ambient light conditions to dark head-mounteddisplay viewing conditions is less abrupt. Similarly, if motion sensordata 90 indicates that device 10 is being removed form a user's head,control circuitry 42 may turn on lighting system 60 so that the user'stransition from dark head-mounted display viewing conditions to brightambient light conditions is less abrupt.

Gaze position 38 may also be used to determine appropriate operatingconditions for lighting system 60 and/or display 40. If the viewer'sgaze is directed towards a bright portion of the image, the user may beslightly bright-adapted, whereas a viewer that is gazing towards a darkportion of the image may be more dark-adapted. Control circuitry 42 may,if desired, determine which light sources 62 in system 60 to illuminateor turn off based on gaze position 38. Gaze position 38 may be measuredusing camera 24 to capture images of the viewer's pupils and otherportions of the viewer's eyes. Gaze position 38 may be used incombination with the pixel luminance at the gaze location to estimatethe user's current adaptation state. This calculation may include, forexample, a moving average over the pixel luminance levels around theuser's gaze position to estimate user's adaptation state. Pixelluminance at the gaze position 38 may be determined by analyzing framesof image data that are being displayed on display 40 (e.g., by analyzingcontent 44 and/or remapped content 50).

During operation, content generators 12 may produce content to bedisplayed on display 40. Content generators 12 may, for example,generate virtual reality content, augmented reality content, and/ormixed reality content. This may include rendering game images in a videogame, retrieving stored movie data and providing corresponding videoframes to be displayed on display 40, producing still image framesassociated with an operating system function or application program,and/or producing other content for displaying on display 40.

Control circuitry 42 may use information on ambient conditions 34,physiological attributes 36, motion sensor data 90, gaze position 38,and/or content 44 and 50 to determine operating conditions for lightingsystem 60 and/or display 40. For example, control circuitry 42 maydetermine a brightness level and color temperature for eachlight-emitting device 62 in lighting system 60 based on one or more ofambient conditions 34, physiological attributes 36, motion sensor data90, gaze position 38, content 44, and content 50.

Tone mapping circuitry 14 in control circuitry 42 may determine howoriginal content values should be mapped to display content values(e.g., to determine how to map content luminance values to displayluminance values in accordance with mapping curves of the type describedin connection with FIG. 2) based on one or more of ambient conditions34, physiological attributes 36, motion sensor data 90, gaze position38, content 44, and content 50. This is sometimes referred to asadjusting the brightness level of the display. To ensure that content isdisplayed appropriately on display 40, tone mapping circuitry 14 canprovide content generators 12 with tone mapping parameters to use inperforming luminance mapping operations and/or tone mapping circuitry 14can implement luminance mapping for content generators 12.

In some configurations, content generators 12 may be capable ofadjusting content luminance values internally. In these situations, tonemapping circuitry 14 can supply content generators 12 with tone mappingparameters. The tone mapping parameters inform content generators 12 ofan appropriate mapping curve to use in supplying content 44 to display40.

In other configurations, content generators 12 may not be capable ofadjusting content luminance values internally or it may otherwise bedesirable to implement tone mapping separately from the tone mappingfunctions of content generators 12. In these circumstances, content 44from content generator 12 may be provided to tone mapping circuitry 14.Tone mapping circuitry 14 may then apply a desiredcontent-luminance-to-display-luminance mapping (e.g., a mapping defineddone mapping parameters associated with one of the curves shown in FIG.2) to ensure that the luminance of content 44 is adjusted appropriately(e.g., so that content 44 is remapped in accordance with a desiredcontent-luminance-to-display luminance mapping to produce correspondingremapped content 50 for displaying on display 40).

FIG. 6 is a graph showing illustrative brightness levels of lightingsystem 60 and display 40 when a user places device 10 on his or her headat time t0. Curve 72 represents the brightness of lighting system 60,and curves 74, 76, and 78 represent three illustrative brightness curvesfor display 40. Control circuitry 42 may use ambient light sensor 22 todetermine an ambient light brightness. The ambient light brightness may,for example, be equal to or close to luminance level L2. Controlcircuitry 42 may therefore illuminate lighting system 60 at luminancelevel L2 at t0 to match the user's adaptation state when the user firstputs device 10 on his or her head at time t0. From time t0 to time t1,control circuitry 42 may gradually decrease the brightness of lightingsystem 60 from L2 to L1. L1 may, for example, be the maximum brightnessof display 40. From time t1 to time t2, control circuitry 42 maygradually decrease the brightness of lighting system 60 from L1 to zero.This allows the user to gradually become dark-adapted so that display 40appears sufficiently bright from time t2 on.

In some arrangements, display 40 may follow brightness curve 74.Brightness curve 74 starts at luminance level L1 at time t0 and remainsat L1. The user's brightness adaptation level from time t0 to time t1may not be as low as brightness curve 74, but the presence of additionalillumination from lighting system 40 may help gradually adjust theuser's adaptation state so that the user's transition from brightness L2to L1 is less abrupt.

In other arrangements, display 40 may follow brightness curve 76.Brightness curve 76 starts at zero brightness at time t0, graduallyincreases to brightness level L1 at time t1, and remains at L1 from t1on. The user's brightness adaptation level from time t0 to time t1 maynot be as low as brightness curve 76, but the presence of additionalillumination from lighting system 40 may help gradually adjust theuser's adaptation state so that the user's transition from brightness L2to L1 is less abrupt.

In other arrangements, display 40 may follow brightness curve 78.Brightness curve 78 remains at zero until time t2. Just when lightingsystem 60 is turned off at time t2, display 40 may be turned on and maygradually increase in brightness from zero at time t2 to L1 at time t3.With this type of arrangement, the user may be fully dark-adapted by thetime display 40 is turned on.

The examples of FIG. 6 are merely illustrative. If desired, lightingsystem 60 and/or display 40 may follow other brightness curves.

Control circuitry 42 may determine operating conditions for display 40and lighting system 60 based on measured ambient light conditions 34(FIG. 5) from ambient light sensor 22 (FIG. 1). Control circuitry 42 maygather one or more ambient light measurements just prior to display 40being turned on, in response to display 40 being turned on, or inresponse to any other suitable action (e.g., in response to motionsensor data 90 indicating that device 10 is being placed on a user'shead). In other arrangements, control circuitry 42 may use predeterminedbrightness curves for display 40 and/or lighting system 60 that shiftover time from bright-adapted to dark-adapted. Predetermined brightnesscurves for display 40 and/or lighting system 60 may, for example, bebased on user studies or other information (e.g., curves 72, 74, 76, and78 may be based on a typical adaptation shift for users that transitionfrom bright indoor lighting to dim head-mounted display lighting).

In some arrangements, it may be desirable to drive or guide a user'sadaptation state from one level to another level. For example, when auser is transitioning out of a head-mounted display viewing experience(e.g. when a movie or video ends, when a user enters a home screen orotherwise indicates the viewing experience is nearing an end, etc.), itmay be desirable to direct the user's adaptation level from dark tobright so that the user does not experience dazzle or discomfort uponremoving device 10 from his or her eyes.

This type of scenario is illustrated in FIG. 7. In this example, curve80 is an illustrative brightness curve for display 40, and curves 82 and84 are illustrative brightness curves for lighting system 60. From timet0 to time t1, the user is viewing content on display 40 and isdark-adapted at adaptation state L1. Content on display 40 is optimizedfor adaptation state L1. At time t1, information from input-outputdevices 18 and/or control circuitry 42 may indicate that the viewingexperience is over or coming to an end. This may include, for example,user input that causes display 40 to pause or stop video, user inputthat results in a home screen being displayed on display 40, controlcircuitry 42 indicating a movie or other video has ended, or otherinformation indicating that the viewing experience is over or nearlyover. From time t1 to time t2, the brightness of display 40 may decreaseto zero when the display is turned off.

In some arrangements, lighting system 60 may follow curve 82. In thistype of arrangement, lighting system 60 is turned on at time t1 (e.g.,in response to information from input-output devices 18 and/or controlcircuitry 42 indicating that the viewing experience is over or coming toan end). The brightness of lighting system 60 may gradually increasefrom zero at time t1 to L2 at time t2. This may ensure that the user isbright-adapted before removing device 10 and encountering bright ambientlight.

In some arrangements, lighting system 60 may follow curve 84. In thistype of arrangement, lighting system 60 is turned on at time t2, justwhen display 40 is turned off. The brightness of lighting system 60 maygradually increase from zero at time t2 to L2 at time t3. This mayensure that the user is bright-adapted before removing device 10 andencountering bright ambient light.

As described above, one aspect of the present technology is thegathering and use of data available from various sources to improve thedisplaying of content. The present disclosure contemplates that in someinstances, this gathered data may include personal information data thatuniquely identifies or can be used to contact or locate a specificperson. Such personal information data can include demographic data,location-based data, telephone numbers, email addresses, home addresses,or any other identifying information.

The present disclosure recognizes that the use of such personalinformation data, in the present technology, can be used to the benefitof users. For example, the personal information data can be used togradually transition from bright ambient lighting conditions to darkambient lighting conditions and vice versa. Accordingly, use of suchpersonal information data allows for a more comfortable viewingexperience. Further, other uses for personal information data thatbenefit the user are also contemplated by the present disclosure.

The present disclosure further contemplates that the entitiesresponsible for the collection, analysis, disclosure, transfer, storage,or other use of such personal information data will comply withwell-established privacy policies and/or privacy practices. Inparticular, such entities should implement and consistently use privacypolicies and practices that are generally recognized as meeting orexceeding industry or governmental requirements for maintaining personalinformation data private and secure. For example, personal informationfrom users should be collected for legitimate and reasonable uses of theentity and not shared or sold outside of those legitimate uses. Further,such collection should occur only after receiving the informed consentof the users. Additionally, such entities would take any needed stepsfor safeguarding and securing access to such personal information dataand ensuring that others with access to the personal information dataadhere to their privacy policies and procedures. Further, such entitiescan subject themselves to evaluation by third parties to certify theiradherence to widely accepted privacy policies and practices.

Despite the foregoing, the present disclosure also contemplatesembodiments in which users selectively block the use of, or access to,personal information data. That is, the present disclosure contemplatesthat hardware and/or software elements can be provided to prevent orblock access to such personal information data. For example, in the caseof gathering information about physiological attributes of the user, thepresent technology can be configured to allow users to select to “optin” or “opt out” of participation in the collection of personalinformation data during registration for services. In another example,users can select not to provide physiological data. In yet anotherexample, users can select to not provide precise location information,but permit the transfer of location zone information.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data. For example, physiologicalinformation can be inferred based on non-personal information data or abare minimum amount of personal information, or based on publicallyavailable information.

The present disclosure also contemplates that head-mounted devices maybe used to display different types of content including virtual realitycontent, augmented reality content, and mixed reality content. In somecases, the displayed content may be merged with a physical environment.

A physical environment refers to a physical world that people can senseand/or interact with without aid of electronic systems. Physicalenvironments, such as a physical park, include physical articles, suchas physical trees, physical buildings, and physical people. People candirectly sense and/or interact with the physical environment, such asthrough sight, touch, hearing, taste, and smell.

In contrast, a computer-generated reality (CGR) environment refers to awholly or partially simulated environment that people sense and/orinteract with via an electronic system. In CGR, a subset of a person'sphysical motions, or representations thereof, are tracked, and, inresponse, one or more characteristics of one or more virtual objectssimulated in the CGR environment are adjusted in a manner that comportswith at least one law of physics. For example, a CGR system may detect aperson's head turning and, in response, adjust graphical content and anacoustic field presented to the person in a manner similar to how suchviews and sounds would change in a physical environment. In somesituations (e.g., for accessibility reasons), adjustments tocharacteristic(s) of virtual object(s) in a CGR environment may be madein response to representations of physical motions (e.g., vocalcommands).

A person may sense and/or interact with a CGR object using any one oftheir senses, including sight, sound, touch, taste, and smell. Forexample, a person may sense and/or interact with audio objects thatcreate 3D or spatial audio environment that provides the perception ofpoint audio sources in 3D space. In another example, audio objects mayenable audio transparency, which selectively incorporates ambient soundsfrom the physical environment with or without computer-generated audio.In some CGR environments, a person may sense and/or interact only withaudio objects.

A virtual reality (VR) environment refers to a simulated environmentthat is designed to be based entirely on computer-generated sensoryinputs for one or more senses. A VR environment comprises a plurality ofvirtual objects with which a person may sense and/or interact. Forexample, computer-generated imagery of trees, buildings, and avatarsrepresenting people are examples of virtual objects. A person may senseand/or interact with virtual objects in the VR environment through asimulation of the person's presence within the computer-generatedenvironment, and/or through a simulation of a subset of the person'sphysical movements within the computer-generated environment.

In contrast to a VR environment, which is designed to be based entirelyon computer-generated sensory inputs, a mixed reality (MR) environmentrefers to a simulated environment that is designed to incorporatesensory inputs from the physical environment, or a representationthereof, in addition to including computer-generated sensory inputs(e.g., virtual objects). On a virtual-reality (“virtuality”) continuum,a mixed reality environment is anywhere between, but not including, awholly physical environment at one end and virtual reality environmentat the other end.

In some MR environments, computer-generated sensory inputs may respondto changes in sensory inputs from the physical environment. Also, someelectronic systems for presenting an MR environment may track locationand/or orientation with respect to the physical environment to enablevirtual objects to interact with real objects (that is, physicalarticles from the physical environment or representations thereof). Forexample, a system may account for movements so that a virtual treeappears stationery with respect to the physical ground.

Examples of mixed realities include augmented reality and augmentedvirtuality.

An augmented reality (AR) environment refers to a simulated environmentin which one or more virtual objects are superimposed over a physicalenvironment, or a representation thereof. For example, an electronicsystem for presenting an AR environment may have a transparent ortranslucent display through which a person may directly view thephysical environment. The system may be configured to present virtualobjects on the transparent or translucent display, so that a person,using the system, perceives the virtual objects superimposed over thephysical environment. Alternatively, a system may have an opaque displayand one or more imaging sensors that capture images or video of thephysical environment, which are representations of the physicalenvironment. The system composites the images or video with virtualobjects, and presents the composition on the opaque display. A person,using the system, indirectly views the physical environment by way ofthe images or video of the physical environment, and perceives thevirtual objects superimposed over the physical environment. As usedherein, a video of the physical environment shown on an opaque displayis called “pass-through video,” meaning a system uses one or more imagesensor(s) to capture images of the physical environment, and uses thoseimages in presenting the AR environment on the opaque display. Furtheralternatively, a system may have a projection system that projectsvirtual objects into the physical environment, for example, as ahologram or on a physical surface, so that a person, using the system,perceives the virtual objects superimposed over the physicalenvironment.

An augmented reality environment also refers to a simulated environmentin which a representation of a physical environment is transformed bycomputer-generated sensory information. For example, in providingpass-through video, a system may transform one or more sensor images toimpose a select perspective (e.g., viewpoint) different than theperspective captured by the imaging sensors. As another example, arepresentation of a physical environment may be transformed bygraphically modifying (e.g., enlarging) portions thereof, such that themodified portion may be representative but not photorealistic versionsof the originally captured images. As a further example, arepresentation of a physical environment may be transformed bygraphically eliminating or obfuscating portions thereof.

An augmented virtuality (AV) environment refers to a simulatedenvironment in which a virtual or computer generated environmentincorporates one or more sensory inputs from the physical environment.The sensory inputs may be representations of one or more characteristicsof the physical environment. For example, an AV park may have virtualtrees and virtual buildings, but people with faces photorealisticallyreproduced from images taken of physical people. As another example, avirtual object may adopt a shape or color of a physical article imagedby one or more imaging sensors. As a further example, a virtual objectmay adopt shadows consistent with the position of the sun in thephysical environment.

There are many different types of electronic systems that enable aperson to sense and/or interact with various CGR environments. Examplesinclude head mounted systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmounted system may have one or more speaker(s) and an integrated opaquedisplay. Alternatively, a head mounted system may be configured toaccept an external opaque display (e.g., a smartphone). The head mountedsystem may incorporate one or more imaging sensors to capture images orvideo of the physical environment, and/or one or more microphones tocapture audio of the physical environment. Rather than an opaquedisplay, a head mounted system may have a transparent or translucentdisplay. The transparent or translucent display may have a mediumthrough which light representative of images is directed to a person'seyes. The display may utilize digital light projection, OLEDs, LEDs,μLEDs, liquid crystal on silicon, laser scanning light source, or anycombination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In one embodiment, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

The foregoing is merely illustrative and various modifications can bemade to the described embodiments. The foregoing embodiments may beimplemented individually or in any combination.

What is claimed is:
 1. An electronic device configured to be worn on auser's head, comprising: an ambient light sensor that measures anambient light brightness outside of the electronic device; a displaythat generates display content; an optical system through which thedisplay content is viewable; a lighting system that providesillumination around a periphery of the optical system; and controlcircuitry that adjusts a brightness of the illumination based on theambient light brightness, wherein the control circuitry reduces thebrightness of the illumination when the display begins generatingdisplay content.
 2. The electronic device defined in claim 1 wherein thelighting system comprises light-emitting diodes.
 3. The electronicdevice defined in claim 2 wherein the light-emitting diodes aredistributed in a loop that extends around the periphery of the opticalsystem.
 4. The electronic device defined in claim 2 wherein thelight-emitting diodes comprise red, green, and blue light-emittingdiodes.
 5. The electronic device defined in claim 4 wherein the ambientlight sensor measures an ambient light color and wherein the controlcircuitry adjusts a color of the illumination based on the ambient lightcolor.
 6. The electronic device defined in 4 wherein the controlcircuitry adjusts a color of the illumination based on the displaycontent.
 7. The electronic device defined in claim 1 wherein thelighting system comprises a light guide and a light source that emitslight into the light guide.
 8. The electronic device defined in claim 7wherein the light guide forms a loop that extends around the peripheryof the optical system.
 9. The electronic device defined in claim 8wherein the light guide has light extraction features that allow thelight to escape from the light guide to provide the illumination. 10.The electronic device defined in claim 1 further comprising a motionsensor that produces motion sensor data, wherein the control circuitryadjusts the brightness of the illumination based on the motion sensordata.
 11. An electronic device configured to be worn on a user's head,comprising: a display that generates display content; an optical systemthrough which the display content is viewable; light sources adjacent tothe optical system; and control circuitry that turns on the lightsources before turning on the display.
 12. The electronic device definedin claim 11 wherein the optical system comprises lenses and wherein thelight sources are distributed in a loop surrounding the lenses.
 13. Theelectronic device defined in claim 12 wherein the light sources compriselight-emitting diodes.
 14. The electronic device defined in claim 11wherein the control circuitry turns off the light sources when thedisplay reaches a given brightness level.
 15. The electronic devicedefined in claim 11 wherein the light sources are configured toilluminate the user's peripheral vision when the electronic device isworn on the user's head.
 16. A method for operating an electronic deviceconfigured to be worn on a user's head, wherein the electronic devicecomprises a display that displays virtual reality content and an opticalsystem through which the virtual reality content is viewable, the methodcomprising: with an ambient light sensor, measuring ambient lightbrightness outside of the electronic device; with a light source,illuminating a periphery of the optical system while the display ispowered off; and with control circuitry, determining a brightness of thelight source based on the ambient light brightness.
 17. The methoddefined in claim 16, further comprising: decreasing the brightness ofthe light source from a first brightness level to a second brightnesslevel.
 18. The method defined in claim 17, further comprising: when thebrightness of the light source reaches the second brightness level,turning on the display.
 19. The method defined in claim 18, furthercomprising: after turning on the display, turning off the light source.20. The method defined in claim 19, further comprising: increasing thebrightness of the light source before the electronic device is removedfrom the user's head.